Snapshot Judgements: Obtaining Data Insights without Tracing

Metadata snapshots are a favored method for gaining filesystem insights due to their small size and relative ease of acquisition compared to access traces [2]. Since snapshots do not include an access history; typically they are used for relatively simple analyses such as file lifetime and size distributions, and researchers still gather and store full block or file access traces for any higher level analysis such as cache prediction or scheduling variable replication [1, 3]. We claim that one can gain rich insights into file system and user behavior by clustering metadata snapshots and comparing the entropy within clusters to the entropy within natural partitions such as directory hierarchies or single attributes. We have preliminary results indicating that agglomerative clustering methods produce groups of data with high information purity, which may be a sign of functional correlation. While many studies have analyzed metadata snapshots, most focus on simple statistics, such as file size, age, or extension, or they attempt to reconstruct dynamic trace information from a series of snapshots by interpolating inter-snapshot accesses. We focus instead on what can be learned about a system by looking at metadata correlations within a small set of widely spaced snapshots. For example, timestamps can give insight into the dynamic activity of the system from a purely static viewpoint. UIDs can be used in conjunction with file paths to figure out if there is a “typical” namespace structure users create. Entropy between members of a namespace can help us relate different segments of a trace [4]. Full I/O traces are always superior, but keeping complete logs of accesses is prohibitive in many systems because of the computational overhead to collect the logs and the storage overhead to keep them. For a modern storage system with hundreds of thousands of I/Os per second, storing even minimal representations of the I/O without any metadata is very costly. For example, an enterprise storage system may create over 16 GB of blocklevel I/O logs per day [5]. Moreover, storing complete traces with metadata is even harder than storing raw accesses because there is more overhead both in terms of size and performance, thus this information is usually lost. We examined a series of clusterings using HPC and Figure 1: Sample clusterings view for a single snapshot. Clusters are indicated by shape and modification time is indicated by color.